Image processing apparatus and method, image synthesizing system and method, image synthesizer and client computer which constitute image synthesizing system, and image separating method

ABSTRACT

Alternatively, a set of pixel image data and a set of coordinate data are respectively compressed, and compressed pixel image data and the compressed coordinate data are stored. To perform proper color correction (conversion) even when a user image is inlaid in a template image to produce a composite image. A position, in which the user image is to be inlaid, on the template image is defined by mask information. The template image and the mask information are correlated with each other. The user image is inlaid in the position, on the template image, defined by the mask information, to produce a composite image. Data representing the produced composite image and the corresponding mask information are transmitted from a client computer to an image synthesizer. In the image synthesizer, the mask information is referred to, to separate the template image and the user image from the composite image. The template image and the user image, which have been separated from each other, are individually subjected to color correction in the image synthesizer. The template image and the user image, which have been respectively subjected to the color correction, are resynthesized.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus andmethod, particularly to an apparatus for and a method of extracting(cutting out) a desired image portion from an original image displayedon a screen of a display device, and an apparatus for and a method offurther extracting a part of the partial image, moving the part of theimage, and arranging (editing) or storing a plurality of partial imagesupon correlating with each other.

Further, the present invention relates to an image synthesizing systemcomprising a client computer and an image synthesizer which cancommunicate data to each other and an image synthesizing method in theimage synthesizing system, the client computer and the image synthesizerwhich constitute the image synthesizing system, and an image separatingmethod.

2. Description of the Background Art

The format of image data includes GIF (Graphic Interchange Format), EPSF(Encapsulated PostScript Format), and so forth.

A color is expressed using a color palette (e.g. of 256 colors) in theGIF. A desired part of an image (a partial image) can be specified inthe original image of the GIF format. The contour of the partial imagecan be designated as desired. An area (background) other than thespecified partial image in the original image is represented by a set ofimage data of a transparent color. The GIF format has such inconveniencethat the specified partial image and the background image are alwaystreated as an integrated image data. Further, an image which isexpressed using colors close to colors in nature (e.g., 16,000,000colors) (referred to as a natural image or a picture of a scene) cannotbe represented by the GIF format.

The EPSF is suited for graphics and graphics is expressed by a vector.Since a picture of a scene hardly be expressed by a vector, the pictureof a scene cannot be also handled in the EPSF format.

On the other hand, when a subject image (a user image) picked up by astill video camera, a movie video camera or the like is fetched into acomputer, and a color image is printed using a color printer, image datarepresenting the user image is subjected to color correction dependingon the printing characteristics of the color printer.

With the development of the computer, it has been possible for a userhimself or herself to inlay the user image, in a position, in which auser image is to be inlaid, on a template image representing thebackground of the user image. A composite image obtained by inlaying theuser image in the template image can be also printed using a colorprinter that the user has.

Even in such a case, the characteristics of the color printer is knownby the user, and the user himself or herself stores in the computerimage data representing the user image and image data representing thetemplate image. Therefore, it is possible to separately take out theuser image data and the template image data, and separately subject theuser image data and the template image data to color correctiondepending on the characteristics of the color printer prior to thesynthesis of the user image and the template image.

The user image data which has been subjected to the color correction andthe template image data which has been subjected to the color conversionare synthesized to produce the composite image data, so that thecomposite image obtained by the printing exhibits proper colors as awhole.

Consider an image synthesizing system comprising a client computer andan image synthesizer located at a place spaced apart from the clientcomputer. In such a system, when a template image and a user image aresynthesized as described above in the client computer, image datarepresenting a composite image is transmitted from the client computerto the image synthesizer, and the composite image is printed using acolor printer connected to the image synthesizer, it is difficult toseparate the template image and the user image from the composite imagein the image synthesizer. Since it is difficult to separate the templateimage and the user image from the composite image it is difficult toindividually perform the most suitable color conversion (inclusive ofcolor correction) conforming to the template image and the most suitablecolor conversion (inclusive of color correction) conforming to the userimage in the image synthesizer.

When the whole of the composite image is subjected to color conversion,even if the color conversion is most suitable for the template image,the color conversion may be, in some cases, unfavorable for the userimage, or vice versa.

SUMMARY OF THE INVENTION

An object of the present invention is to make it possible to handle,when a desired partial image having a desired shape is extracted (cutout) from an image, only the extracted partial image, i.e.,independently from a background image (the remaining image portion).

Another object of the present invention is to make it possible todesignate and extract (cut out) a partial image from not only a naturalimage (a picture of a scene) but also an image represented using a colorpalette.

Still another object of the present invention is to make it possible tofurther extract a part of the extracted partial image to obtain aplurality of partial images, and to define the positional relationshipamong the plurality of the partial images obtained by the extraction.

Still further object of the present invention is to make it possible toperform, even after a template image has been inlaid in a user image toproduce a composite image, color conversion suitable for the templateimage and the user image individually.

The first invention is directed to an image processing apparatus, whichcomprises an image area designating device for designating an area to beextracted of an original image displayed on a display screen, colorchanging means for changing the color of an area, excluding the area tobe extracted which has been designated by the image area designatingdevice, of the displayed original image into a particular color, andstorage controlling means for storing pixel data representing pixelshaving colors excluding the particular color obtained by the change inthe color changing means in the displayed original image and coordinatedata representing positions of the pixels in the displayed originalimage with the pixel data and the coordinate data correlated with eachother.

The first invention provides an image processing method which comprisesthe steps of designating an area to be extracted of an original imagedisplayed on a display screen, changing the color of an area, excludingthe designated area to be extracted, of the displayed original imageinto a particular color, and storing pixel data representing pixelshaving colors excluding the particular color obtained by the change inthe displayed original image and coordinate data representing positionsof the pixels in the displayed original image with the pixel data andthe coordinate data correlated with each other.

The first invention further provides a storage medium storing thereon aprogram for controlling a computer, the program causing the computer toaccept an area to be extracted which is designated on an original imagedisplayed on a display screen, to replace image data representing animage of an area excluding the accepted area to be extracted with imagedata representing a specified color on the image data of the originalimage, to generate pairs of pixel data and coordinate data from theimage data excluding the image data representing the specified color,and to store the generated pairs of the pixel data and the coordinatedata on a storage medium.

According to the first invention, since the image data in the area(background area) other than the area which has been specified to beextracted (cut out) are replaced with image data representing theparticular color, the image data of the specified (designated) area canbe easily obtained merely by extracting other image data than the imagedata which represent the particular color.

The pixel data (pixel image data) and the coordinate data are producedbased on the extracted image data and are stored in correlation witheach other. The extracted image data have coordinate data representingthe position of the pixels in the image for each pixel, so that theextracted image data alone, that is, without accompanying image datarepresenting the background, can be treated or handled. The image datarepresenting the background can be dispense with or unnecessary. Thecoordinate data is for specifying the relative positions of the pixelsin image. The coordinate data may be defined using any point on theextracted image (inclusive an area including the extracted image, e.g. arectangle which circumscribes the extracted image) as an origin of thecoordinate.

According to the first invention, it is possible to designate and toextract a partial image from not only an image represented using a colorpalette but also a natural image.

In the preferred embodiment, the pixel data and the coordinate data formpairs to be stored.

In another embodiment, a set of the pixel data is compressed, and a setof the coordinate data is compressed. The compressed pixel data and thecompressed coordinate data are correlated with each other to be stored.The pixel data and the coordinate data are individually subjected todata compression respectively conforming to the characteristics ornature of the pixel data and coordinate data, so that efficient datacompression can be achieved.

The first invention further provides a recording or storage mediumhaving the above image data (pixel data and coordinate data) storedthereon.

One of the image data storage medium according to the first invention isthe storage medium storing thereon pairs of pixel data and coordinatedata representing pixels composing an image in the order represented bythe coordinate data.

Another of the image data storage medium according to the firstinvention is the storage medium storing thereon compressed pixel dataand compressed coordinate data which are obtained by respectivelycompressing such a group of pixel data and a group of coordinate datathat the pixel data and the coordinate data representing pixelscomposing an image are arranged for pixel data and for coordinate datain the order represented by the coordinate data.

An apparatus for reproducing an image represented by the image datawhich has been stored in accordance with the first invention comprisesdata reading means for reading, from a storage medium storing image datacomprising a set of pairs of pixel data representing pixels andcoordinate data representing the positions of the pixels on the image,the pixel data and the coordinate data, and display controlling meansfor controlling a display device such that the image is displayed bydisplaying the pixels represented by the pixel data read by the datareading means in the positions represented by the coordinate data.

A method of reproducing an image comprises the steps of reading, from astorage medium storing image data comprising a set of pairs of pixeldata representing pixels and coordinate data representing the positionsof the pixels on the image, the pixel data and the coordinate data, andcontrolling a display device such that the image is displayed bydisplaying the pixels represented by the read pixel data in thepositions represented by the corresponding read coordinate data.

An apparatus for reproducing an image represented by the compressedimage data which has been prepared and stored in accordance with thefirst invention comprises data reading means for reading, from a storagemedium storing thereon compressed pixel data and compressed coordinatedata which are obtained by compressing a set of pixel data and bycompressing a set of coordinate data in image data having a datastructure such that pixels constituting an image are represented bypixel data and coordinate data, the compressed pixel data and thecompressed coordinate data, pixel data expanding means for expanding thecompressed pixel data read by the data reading means, coordinate dataexpanding means for expanding the compressed coordinate data read by thedata reading means, and display control means for controlling a displaydevice such that the image is displayed by displaying the pixelsrepresented by the pixel data expanded by the pixel data expanding meansin the positions represented by the coordinate data expanded by thecoordinate data expanding means.

A method of reproducing the compressed image data comprises the steps ofreading, from a storage medium storing thereon compressed pixel data andcompressed coordinate data which are obtained by compressing a set ofpixel data and by compressing a set of coordinate data in image datahaving a data structure such that pixels constituting an image arerepresented by pixel data and coordinate data, the compressed pixel dataand the compressed coordinate data, expanding the read compressed pixeldata, expanding the read compressed coordinate data, and controlling adisplay device such that the image is displayed by displaying the pixelsrepresented by the expanded pixel data in the positions represented bythe expanded coordinate data.

The second invention is directed an image processing apparatus, whichcomprises a designating device for designating on an image displayed ona display screen based on image data comprising pixel data representingpixels and coordinate data representing the positions of the pixels, apartial image to be moved, a movement value inputting device forinputting the direction of movement and the amount of movement of thepartial image, movement amount adding means for adding the coordinatevariation corresponding to the amount of movement in the direction ofmovement inputted by the movement value inputting device to thecoordinate data of the pixels composing the partial image designated bythe designating device, and storage controlling means for storing thenew coordinate data obtained by the movement amount adding means and thepixel data corresponding thereto in correlation with each other as wellas the coordinate data and the pixel data representing the image otherthan the partial image.

An image processing method according to the second invention comprisesthe steps of designating on an image displayed on a display screen basedon image data comprising pixel data representing pixels and coordinatedata representing the positions of the pixels, a partial image to bemoved, inputting the direction of movement and the amount of movement ofthe partial image, adding the coordinate variation corresponding to theinputted amount of movement in the inputted direction of movement to thecoordinate data of the pixels composing the designated partial image,and storing the new coordinate data obtained by the addition and thepixel data corresponding thereto in correlation with each other as wellas the coordinate data and the pixel data representing the image otherthan the partial image.

The second invention further provides a storage medium storing thereon aprogram for controlling a computer, the program causing the computer toaccept an area of a partial image designated on an image displayed on adisplay screen based on image data comprising pixel data representingpixels and coordinate data representing the positions of the pixels, toaccept the direction of movement and the amount of movement of thepartial image, to calculate new coordinate data by adding the coordinatevariation corresponding to the accepted amount of movement in theaccepted direction of movement to the coordinate data of the pixelscomposing the partial image of the accepted area, and to store the newcoordinate data obtained by the calculation and the pixel datacorresponding thereto in correlation with each other as well as thecoordinate data and the pixel data representing the image other than thepartial image.

In a preferred embodiment of the second invention, the partial imagebefore movement disappears, and the partial image after movement appearsat the positions defined by the new coordinate data on the screen of thedisplay device. The user can recognize the position of the moved partialimage.

According to the second invention, even in a case where a part of animage is extracted (cut out) and the extracted partial image is moved,the pixel data and the new coordinate data which represent the movedpartial image are correlated with each other to be stored together withthe pixel data and the coordinate data of the original image (exclusiveof the moved partial image).

Preferably, all of the above pixel data and the coordinate data arestored in the predetermined order, e.g., the order represented by thecoordinate data. A set of the pixel data and a set of the coordinatedata are individually compressed, and the compressed pixel data andcompressed coordinate data are combined to be stored.

In this way, in a case where a part of an image is extracted and ismoved, the image data representing these images (the extracted partialimage and the original image exclusive of the extracted partial image)are saved in a state that the positional relationship of these images iskept. It is possible to handle or treat a plurality of images which areseparated from each other with maintaining the mutual positionalrelationship therebetween.

An image synthesizing system according to the third invention comprisesa client computer and an image synthesizer which can communicate data toeach other.

The client computer comprises an image synthesizing device for inlayinga user image or a portion thereof, in a position defined by maskinformation accompanying a template image, on the template imagerepresenting the background of the user image, and a composite imageinformation transmitting device for transmitting composite image datarepresenting a composite image produced by the image synthesizing deviceand the mask information used for the synthesis, upon correlating witheach other, to the image synthesizer.

The image synthesizer comprises a composite image information receivingdevice for receiving the composite image data and the mask informationwhich have been transmitted from the client computer, and an image dataseparating device for separating at least one of image data representingthe template image and image data representing the user image from thecomposite image data representing the composite image, on the basis ofthe received mask information.

The third invention also provides a method suitable for the imagesynthesizing system. That is, the method is an image synthesizing methodin the image synthesizing system comprising the client computer and theimage synthesizer which can communicate data to each other.

In the client computer, a user image or a part thereof is inlaid in aposition defined by mask information accompanying a template image, onthe template image representing the background of the user image, andthe composite image data representing a composite image obtained byinlaying the user image in the template image and the mask informationused for inlaying are correlated with each other and are transmittedfrom the client computer to the image synthesizer.

In the image synthesizer, the composite image data and the maskinformation which are transmitted from the client computer are received,and at least one of image data representing the template image and imagedata representing the user image, which constitute the composite imagedata representing the composite image are separated, from the compositeimage data on the basis of the received mask information.

According to the third invention, the user image is inlaid in theposition defined by the mask information representing the position, inwhich the user image is to be inlaid, on the template image to producethe composite image in the client computer. The composite image datarepresenting the composite image and the corresponding mask informationare correlated with each other, and are transmitted from the clientcomputer to the image synthesizer.

In the image synthesizer, the composite image data and the maskinformation, which have been transmitted from the client computer, arereceived. The position, in which the user image is to be inlaid, on thetemplate image is recognized on the basis of the received maskinformation. Since the position in which the user image is inlaid isfound, the template image data and the user image data can be relativelyeasily separated from the composite image data.

Since the template image data and the user image data, which constitutethe composite image data, are separately obtained, the template imagedata and the user image data can be separately and individuallysubjected to color conversion, as required. It is possible to performthe most suitable color conversion for the template image data and themost suitable color conversion for the user image.

The composite image is produced again from the template image data andthe user image data at least one of which has been subjected to thecolor conversion, as required. A re-composite image thus obtained by theresynthesis is printed in a printer. Since at least one of the templateimage and the user image, which constitute the re-composite image, isindividually subjected to the color conversion, the re-composite imagewhose colors have been converted to proper colors is printed in theprinter.

The foregoing and other object, features, aspects and advantages of thepresent invention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the appearance of an image recording/reproducingapparatus;

FIG. 2 illustrates the outline of the electrical configuration of theimage recording/reproducing apparatus;

FIGS. 3 to 10 illustrate examples of display images;

FIG. 11 illustrates an example of a temporary storage format for anatural image;

FIG. 12 illustrates an example of a temporary storage format for apseudo image;

FIG. 13 illustrates the format of coordinate data which has beensubjected to increment length compression;

FIG. 14 illustrates a format for recording compressed pixel data andcoordinate data for a natural image;

FIG. 15 illustrates a format for recording compressed pixel data andcoordinate data for a pseudo image;

FIG. 16 is a flow chart showing the procedure for recording processingof an edited image;

FIG. 17 shows the procedure for image display processing;

FIG. 18 shows the procedure for image editing processing;

FIG. 19 is a flow chart showing the procedure for image reproductionprocessing;

FIG. 20 illustrates the overall configuration of an image synthesizingsystem;

FIG. 21 is a block diagram showing the electrical configuration of aclient computer;

FIG. 22 is a block diagram showing the electrical configuration of animage synthesizer;

FIG. 23 shows how a composite image is produced;

FIG. 24 illustrates the format of a template image file;

FIG. 25 illustrates the format of a composite image file;

FIG. 26 schematically shows the procedure for processing for printing acomposite image in an image processing system; and

FIG. 27 is a flow chart showing the procedure for processing forprinting a composite image in an image processing system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS (1) First Embodiment

FIG. 1 illustrates the appearance of an image processing(recording/reproducing) apparatus according to the present embodiment.FIG. 2 is a block diagram showing the electrical configuration of theimage processing (recording/reproducing) apparatus.

With reference to these figures, the image processing(recording/reproducing) apparatus comprises a computer 1. A CRT displaydevice 10, an image scanner 11, a keyboard 12 and a mouse 13 areconnected to the computer 1.

The whole of image (recording and reproducing) processing is supervisedby a CPU 2 of the computer 1. An FD (floppy disk) drive 7, a CD-ROM(compact disk read only memory) drive 8 and an HD (hard disk) drive 9are provided inside the computer 1. The FD drive 7 writes data to an FD17 and reads out data from the FD 17. The CD-ROM drive 8 reads out dataand programs from a CD-ROM 18. The HD drive 9 writes data to a hard disk(not shown) and reads out data from the hard disk. Programs making thecomputer 1 perform various processing (FIGS. 16, 17, 18 and 19) is readout of the CD-ROM 18 and installed in the hard disk.

The scanner 11, the keyboard 12 and the mouse 13 are connected to thecomputer 1 by an input/output interface 6. The computer 1 furthercomprises an image storage memory 5 for temporarily storing image datawhen an image is extracted and a display memory 4 for temporarilystoring the image data when the image is displayed on the display device10. The image data stored in the display memory 4 is read out, and isfed to the display device 10 through a display interface 3, so that theimage is displayed.

The image processing (recording/reproducing) apparatus displays adesired image (an original image) on the display device 10, extracts apart of the image (a partial image) from the original image displayed ona screen of the display device 10, edits the extracted partial image,and records on the FD 17 image data representing the edited image. Theimage data representing the partial image extracted from the originalimage may be recorded on the FD 17, as required.

With reference to examples of display images of the display device 10shown in FIG. 3 to 10, processing in the image processing apparatus isexplained in accordance with a flowchart shown in FIG. 16.

Image data representing one or a plurality of original images are storedin the FD 17, the CD-ROM 18 or the hard disk in advance. Image datawhich represent a desired original image is read out from one of abovestorage media by the FD drive 7, the CD-ROM drive 8 or the HD drive 9(step 21). The image data read out is temporarily stored in the displaymemory 4. The original image represented by the image data stored in thedisplay memory 4 is displayed on a display screen of the display device10 as shown in FIG. 3.

It should be understood that FIG. 3 shows the original image displayedon a part of the display screen of the display device 10. Generally anwindow appears on the display screen and the original image is displayedwithin the window. In this case, the rectangular frame within which theoriginal image appears shown in FIG. 3 represents a contour of thewindow.

The original image may be an image appearing on a film, a photograph andother visible media. In this case, the original image on the visiblemedium is read by the scanner 11, and the original image is displayedusing the image data obtained by the scanner 11.

An image area Al to be extracted (cut out) is designated using the mouse13 by a user in a state where the original image is displayed on thedisplay screen of the display device 10 (step 22). That is, the contourof the image area Al to be extracted (cut out) is drawn by moving acursor displayed on the screen using the mouse 13. The drawn contour ismemorized in the display memory 4 in the form of contour image data (theimage data representing the contour is overwritten).

In the display memory 4, the image data of pixels which reside outsidethe contour (within the window, of course) on the image are replacedwith image data representing a specified color (the image datarepresenting the specified color are overwritten). Preferably, thespecified color is a color which does not exist in a nature image (apicture of a scene) or a color probability of which is very low (forexample, a color the primary color (R, G, B) data of which arerepresented by such data as FF, FE, 00, 01 or combination thereof). Inthis way, the color of an image area A2 (of course, the area A2 islimited within the window) outside the image area A1 to be extractedwhich has been designated using mouse 13 is changed into thepredetermined and specified (particular) color as shown in FIG. 4. Inthis drawing, the image area A2 the color of which has been changed intothe specified color is hatched for easy understanding.

The image data within the window are scanned on the display memory 4 asshown in FIG. 5, and the image data residing within the area A1 to beextracted are read out of the display memory 4. The read-out image dataare temporarily stored in the hard disk in accordance with apredetermined temporary storage format in such form that pixel data(image data representing individual pixel is referred to as the “pixeldata”) and data representing a coordinate position (coordinate data) ofthe pixel are correlated with each other (step 23). Since the image dataof the area A2 outside the area A1 to be extracted are the datarepresenting the specified color as described above, the image datawithin the area A1 can be read out by discriminating whether the data isone other than the data representing the specified color. The coordinateof the pixel data is determined with a specified point P of the window(e.g. left upper corner) serving as an origin. The coordinate data andthe address of the display memory 4 are related to each other inone-to-one correspondence.

FIG. 11 shows an example of the temporary storage format, which is forthe natural image (scene) represented by 24 bits image data. Thetemporary storage format includes a header recording area and a datarecording area.

The header recording area includes a header size representing the amountof data recorded on a header, a length size and a breadth sizerespectively representing the length and the breadth of the originalimage (see FIG. 3), an image type indicating which of a color image anda monochrome image is the image, a bit depth indicating how many bitscompose respective R, G and B data of pixel data, a coordinate bit depthindicating how many bits compose coordinate data, presence or absence ofa color palette, and a data size representing the amount of datarecorded on the data recording area.

Data relating to each pixel is arranged in the order of above scanningin the data recording area. The data relating to each pixel comprisesthe pixel data which is the image data for each pixel and the coordinatedata which indicates the position of the pixel in the original image. Inthe data relating to each pixel, the pixel data is followed by thecoordinate data so as to relate the pixel data and the coordinate datawith each other. The pixel data comprises R (red), G (green) and B(blue) data, which are arranged in this order and each of which iscomposed of eight bits. Therefore, approximately 16,000,000(=256×256×256) colors can be represented, so that a picture of a scene(natural image) can be expressed. The coordinate data comprises the Xcoordinate data and the Y coordinate data, which are arranged in thisorder. The X coordinate data and the Y coordinate data are respectivelycomposed of 16 bits.

In the temporary storage format described above, each of the pixel datauses a total of 24 bits for R, G, and B. The amount of the pixel datacan be reduced using a color palette. A temporary storage format for thecolor palette (pseudo picture) is illustrated in FIG. 12.

The header recording area of the format shown in FIG. 12 is the same asthat of the format of FIG. 11. The data recording area includes thereina color palette recording area in which image data representing 256types of color are recorded. The data relating to each pixel andrecorded in the data recording area includes a color index, the Xcoordinate data and Y coordinate data which are recorded in this order.The color index designates one of 256 colors in the color palette.

With reference to FIG. 16 again, the pixel data and the coordinate datawhich have been temporarily stored in the hard disk in accordance withthe temporary storage format are read out, and an image to be edited I3is displayed, as shown in FIG. 6, on the display screen of the displaydevice 10 on the basis of the pixel data and the coordinate data whichhave been read out (by the pixel data which have been written into thedisplay memory 4, in accordance with the coordinate data) (step 24). Theimage I3 is the same as the image A1 which has been extracted and isdisplayed such that the image I3 appears in the window. The details ofthe display processing of the image to be edited I3 will be describedlater.

When the image to be edited I3 is displayed on the display screen, imageediting processing (further extraction of another partial image from theimage I3 and movement of the extracted partial image) is performed bythe user (step 25). The details of the image editing processing will bealso described later.

The edited image I4 is shown in FIG. 9. When the image editingprocessing is terminated, image data representing an edited image I4 aretemporarily stored in the hard disk in accordance with the temporarystorage format described above (step 26). The coordinate data stored inthe hard disk in accordance with the temporary storage format are readout, and a coordinate conversion of the coordinate data is performed(step 27, see FIGS. 9 and 10). The image the coordinate data of whichhave been converted is shown in FIG. 10.

The coordinate conversion is performed as follows: A rectangle Rcircumscribing the edited image I4 is assumed. The assumed rectangle Ris not displayed on the screen. An appropriate point Q on the rectangleR (for example, a point of left and upper corner) is selected to be anorigin of a new coordinate system. The coordinate conversion from the XYcoordinate system with the point P serving as the origin to the new XYcoordinate system with the origin Q is performed for each of the pixelscomposing the edited image I4. The converted coordinate data are storedagain in the hard disk in accordance with the temporary storage formatupon being correlated with the pixel data.

When the coordinate data conversion is completed, the pixel data and thecoordinate data corresponding to the pixel data are separated from eachother (step 28). That is, all the pixel data of the edited image I4 aregathered and are arranged in the order of the scanning (the same as theorder in the temporary storage format), and all the coordinate data arealso gathered to be arranged in the same order as that of the pixeldata. A set of the pixel data and a set of the coordinate data areproduced.

The set of the pixel data is subjected to data compression, e.g.,Huffman compression (step 30).

The set of the coordinate data is subjected to increment lengthcompression (step 29). The increment length compression includes atwo-dimension to one-dimension conversion processing of thetwo-dimensional coordinate data (X, Y), and a data compressionprocessing of the one-dimensional coordinate data.

The two/one dimension conversion processing is performed through thefollowing equation;

(one-dimensional coordinate value)=(Y coordinate value)×(width ofimage)+(X coordinate value).

The width of image is the width of the assumed rectangle R as shown inFIG. 10.

The data compression processing of the one-dimensional coordinate data(values) comprises a grouping (or clustering) processing for creating aplurality of groups (or clusters) of coordinate values (data) from amongthe set of the coordinate values which are arranged in descending order,while separating at the discontinuous points of the coordinate values(each group comprises a subset of continuos coordinate values; thenumber of the coordinate values included in each group is referred to as“increment length”), the produced groups being arranged in the order ofthe coordinate values belonging to the groups; a difference calculationprocessing for calculating the difference between the head (starting)coordinate value of one group and the head (stating) coordinate value ofanother group which arranged adjacent to said one group; and arrangingthe head coordinate value of the first (starting) group (one-dimensionalhead coordinate value), the increment length of the first group, thedifference between the head coordinate value of the preceding group andthe head coordinate value of the succeeding group (difference betweenone-dimensional head coordinate values), and the increment length of thesucceeding group in this order.

For example, assume that the one-dimensional coordinate values which arearranged in descending order are “24, 25, 26, 27, 28, 128, 129, 130,131, 156, 157, 158, 159”, for example. Four coordinate values arecontinuous from “24” to “28” in the first group (exclusive of the headvalue), three coordinate values are continuous from “128” to “131” inthe second group (exclusive of the head value), and three coordinatevalues are continuous from “156” to “159” in the third group (exclusiveof the head value). The “difference between one-dimensional headcoordinate value” between the first group and the second group, andbetween the second group and the third group are 128−24=104 and156−128=28, respectively. The data obtained by the increment lengthcompression is “24, 4, 104, 3, 28, 3”.

FIG. 13 shows a format of the increment length compression result data.First, the “one-dimensional head coordinate value” and the “incrementlength” of the first group are arranged in this order. Next, the“difference between one-dimensional head coordinate values” between thefirst group and the second group and the “increment length” of thesecond group are arranged in this order. In the similar manner, the“difference between one-dimensional head coordinate values” between thepreceding group and the succeeding group and the “increment length” ofthe succeeding group are repeatedly arranged.

The pixel data, which has been subjected to Huffman compression, and thecoordinate data, which has been subjected to increment lengthcompression, are connected to each other (step 31) i.e., they arearranged such that the compressed pixel data is followed by thecompressed coordinate data. The pixel data and the coordinate data,which have been connected to each other, are recorded on the FD 17 inaccordance with the format shown in FIG. 14 or 15 (step 32).

FIG. 14 shows a compressed data format for the image data of a naturalimage (a picture of a scene). The header recording area are basicallythe same as that shown in FIG. 11, except that the length size and thebreadth size are replaced with the height size and the width size,respectively, and that the data size is replaced with the pixel datasize and the coordinate date size. The data recording area has thecompressed pixel data and the compressed coordinate data arranged andstored therein in this order. FIG. 15 shows a compressed data format forthe image data of the pseudo image. The header recording area is almostthe same as that shown in FIG. 14, except that the pixel data size isreplaced with the index data size. The data recording area includes thecolor palette, the compressed index data and the compressed coordinatedata arranged in this order.

Referring to FIG. 17, description is made of processing (FIG. 16, step24) for displaying an image to be edited.

Pixel data and coordinate data which have been converted into atemporary storage format are read out from the hard disk (steps 41 and42), a pixel represented by the pixel data is displayed at a positiondesignated by the coordinate data by writing the pixel data into thedisplay memory 4 in accordance with the coordinate data (step 43). Theprocessing of steps 41 to 43 are repeatedly executed for the all pixeldata representing the image (step 44). Consequently, the image extractedby the user is displayed on the display screen of the display device 10,as shown in FIG. 6.

Referring to FIGS. 7, 8 and 18, description is made of image editingprocessing (FIG. 16, step 25).

The example of the image editing processing described hereinafter isthat a part of image (a partial image) is further extracted (cut out)from the displayed image to be edited, and the extracted partial imageis moved in an arbitrary direction by an arbitrary distance.

In the image to be edited displayed on the display screen of the displaydevice 10, an area of a partial image A4 to be further extracted fromthe image is designated by the user using the mouse 13 (step 51). Anwindow A5 circumscribing or surrounding the designated partial image A4is set for processing (step 52, see FIG. 7).

The pixel data and the coordinate data representing the partial image A4are read from the hard disk (step 53). The pixel data and the coordinatedata, which have been read, are temporarily stored in the image storagememory 5 (step 54).

The amount of movement and the direction of movement of the partialimage A4 are set by the user (step 55). For example, a cursor on thedisplay screen is moved in a desired direction by a desired distanceusing the keyboard 12 or the mouse 13. The amount and the direction ofthe movement of the cursor is read in the computer 1. The variation Δxand Δy corresponding to the amount and the direction of the movement ofthe cursor are respectively added to the coordinate values X and Y ofthe pixels which compose the partial image A4 to obtain new coordinatedata (values) for the partial image A4 (step 56).

The window A5 is enlarged such that the window A6 surrounds andcircumscribes the moved partial image A4 (step 57). Within the enlargedwindow A6, the pixel data representing the partial image A4 beforemovement are erased and the pixel data representing the partial image A4after movement are overwritten into the display memory 4 in accordancewith the new coordinate data. Thus, the partial image A4 after movementis displayed in the display screen (step 58, see FIG. 8).

FIG. 19 shows the procedure for processing for reproducing an imagebased on the data which have been recorded on the FD 17 in accordancewith the format shown in FIG. 14 or FIG. 15.

The compressed pixel data and the corresponding coordinate data are readout of the FD 17 (step 61), and are separated from each other (step 62).

The compressed pixel data is expanded by being subjected to Huffmandecoding (step 63). The compressed coordinate data is first convertedinto the one-dimensional coordinate data by being subjected toexpansion. Further, the one-dimensional coordinate data is convertedinto the two-dimensional coordinate data (step 64). In this way, theexpanded pixel data and the expanded coordinate data are obtained.

The expanded pixel data and the expanded coordinate data are temporarilystored in the hard disk in accordance with the temporary storage formatdescribed above (step 65). The pixel data and the coordinate data areread out from the hard disk, and the pixel data are written into thedisplay memory 4 in accordance with the coordinate data, so that theedited image as shown in FIG. 8 or FIG. 10 is displayed on the displaydevice 10 (step 66).

In the above embodiment, the edited image (the pixel data and thecoordinate data thereof) is compressed. Of course, the image data beforeediting (the image data obtained at step 23 of FIG. 16) may becompressed to be stored or recorded. The compressed data may be storedin the hard disk instead of the FD 17.

In the coordinate conversion processing at step 27 of FIG. 16, theappropriate point Q on the rectangle R which circumscribes the partialimage I4 is adopted as a coordinate origin. In order to device thecoordinate origin, an rectangle which circumscribes a part of the imageI4 may be considered. Alternatively, a particular point within thepartial image I4 may be adopted as a coordinate origin. The coordinateconversion processing may be executed before the editing processing. Thecoordinate conversion is not necessarily required.

(2) Second Embodiment

FIG. 20 illustrates the overall configuration of an image synthesizingsystem according to the second embodiment.

The image synthesizing system comprises a client computer 71 and animage synthesizer 100 which can communicate image data and other data toeach other via a network. A color printer 90A is connected to the imagesynthesizer 100.

In the image synthesizing system, a user image that a user of the clientcomputer 71 has is inlaid in a template image in the client computer 71,to produce a composite image. The produced composite image and maskinformation used for producing the composite image and representing aposition, in which the user image is to be inlaid, are transmitted fromthe client computer 71 to the image synthesizer 100. In the imagesynthesizer 100, the template image and the user image, which constitutethe composite image, are separated from each other on the basis of themask information.

The user image and the template image, which have been separated fromeach other, can be separately subjected to color conversion (colorcorrection), as required, for example, depending on the characteristicsof the color printer 90A. The template image and the user image, atleast one of which has been subjected to the color conversion (colorcorrection), are resynthesized, to produce a composite image, so thatthe obtained final composite image (re-composite image) has propercolors.

FIG. 21 is a block diagram showing the electrical configuration of theclient computer 71. The client computer 71 is generally placed in auser's home.

The overall operations of the client computer 71 are supervised by a CPU72.

The client computer 71 comprises a ROM 73, a RAM 74 for temporarilystoring data, a VRAM 82 for temporarily storing data representing animage displayed on a display device 84 so as to edit an image, and amemory controller 85 for controlling the reading of data from the ROM 73and the writing and reading of data to and from the RAM 74 and the VRAM82. Image data read out from the VRAM 82 is fed to a digital-to-analog(DA) converter 83, where it is converted into an analog image signal.Consequently, an image is displayed on the display device 84.

A bus controller 75, the memory controller 85, and a timer 86 areconnected to the client computer 71.

Furthermore, a system I/O controller 76 is connected to the clientcomputer 71. A keyboard 77 and a mouse 78 for accepting an operationinstruction from the user of the client computer 71, a CD-ROM drive 79and an FD drive 80 for reading image data, and a modem 81 for makingconnection to the network are connected to the system I/O controller 76.

An external I/O controller 88 is connected to the client computer 71. Aflat bet scanner 91, a film scanner 92, a digital still camera 93, andan HD drive 94 are connected to the external I/O controller 88. Aprogram for controlling the client computer 71 is stored in an HD (ahard disk; not shown), to and from which data (inclusive of programs)are read and written by the HD drive 94. (The program is installed inthe client computer 71 from the CD-ROM and stored in the HD. A CD-ROMstoring the program is loaded in the CD-ROM drive 79 and the program isread by the CD-ROM drive 79.) The program stored in the HD is read out,so that predetermined processing, described later, is performed by theclient computer 71.

A printer 90 for printing an image and a printer control circuit 89 forcontrolling the printer 90 are further connected to the client computer71.

The user inlays the user image in the template image using the clientcomputer 71.

FIG. 22 is a block diagram showing the electrical configuration of theimage synthesizer 100. A high-quality color printer 90A is provided andconnected to the image synthesizer 100. In FIG. 22, the same referencenumerals are assigned to the same components as those shown in FIG. 21and hence, the description thereof is not repeated.

A program for controlling the image synthesizer 100 is also installed inthe image synthesizer 100 from the CD-ROM and stored in an HD. A CD-ROMstoring the program is loaded in a CD-ROM drive 79 and the program isread by the CD-ROM drive 79.

A high-speed film scanner 95 is connected to an external I/O controller88 included in the image synthesizer 100. Various image files andholders are stored in the HD, to and from which data are written andread by the HD drive 94 connected to the external I/O controller 88.

In the present embodiment, the user of the client computer 71 has a userimage. Examples of the user image are a visible image such as aphotograph, a film and so on, and images represented by image datarecorded on a memory card, a floppy disk (FD), a compact disk (CD) andthe like. In a case where the user image is the photograph, thephotograph is read by the flat bed scanner 91 to obtain image datarepresenting the user image. In a case where the user image is the film,the film is read by the film scanner 92 so that image data of the userimage is produced. In a case where the user image is represented by theimage data recorded in the memory card, the image data is read from thememory card by the digital still camera 93. In a case where the userimage is represented by the image data which has been recorded on theFD, the image data representing the user image is read from the FD bythe FD drive 80. In a case where the user image data is recorded on theCD-ROM, the image data is read by the CD-ROM drive 79 from the CD-ROM.

FIG. 23 is a perspective view showing how a composite image is produced.

A template image is an image representing the background of the userimage. A position, in which the user image is to be inlaid, on thetemplate image is defined by a mask image. A window Aw for defining theposition, in which the user image is to be inlaid, on the template imageis determined in the mask image. The mask image is stored together withthe corresponding template image as mask information in the same file,so that the template image and the mask image are correlated with eachother.

The template image and the mask image may be produced in the clientcomputer 71 by the user or may be obtained from another computer inwhich the template image and the mask image are produced. In eithercase, template image data representing the template image and the maskimage data representing the mask image are recorded on the HD.

The image synthesizing processing is performed as follows.

In the mask image, all the pixels within the window Aw is represented bydata “1” and all the pixels within the other area (masking area hatchedin FIG. 23) than the window Aw is represented by data “0”. AND logicbetween the user image data and the mask image data is operated toextract the part of the user image which just corresponds to the windowAw. NAND logic operation between the mask image data and the templateimage data results in the template image having a blank area which justcorresponds to the window Aw. The composite image is obtained by ORlogic between the partial user image corresponding to the window Aw andthe template image lacking the data corresponding to the window Aw.

FIG. 24 illustrates the format of the template image file.

The template image file stores a header indicating that this file is atemplate image file, template image data representing the templateimage, and mask information defining the window Aw in the mask imagecorresponding to the template image. The mask information can berealized by various embodiments. The mask information may be the imagedata of the mask image, may be the image data representing only thewindow Aw, and may be a set of codes representing a shape of the windowand feature points defining the size and position of the shape (e.g.,coordinate of a center of a circle, coordinates of two vertices of arectangle). When the template image data is read, the mask informationcorresponding to the template image data is also read.

FIG. 25 illustrates the format of the composite image file produced inthe client computer 71. The composite image file stores a headerindicating that the file itself is a composite image file, compositeimage data obtained in the image synthesis processing, and maskinformation used when the composite image is constructed. When thecomposite image data is read from the composite image file, the maskinformation corresponding to the composite image data is also read.

FIG. 26 schematically illustrates the procedure for image synthesisprocessing in the image synthesizing system, and FIG. 27 is a flow chartshowing the procedure for the image synthesis processing.

Referring mainly to these figures, description is made of processingperformed when images are synthesized, separated and resynthesized inthe image synthesizing system.

In the client computer 71, the template image data and the maskinformation are created. The created template image data and the maskinformation are recorded on the HD of the client computer 71 (step 121).When the template image data and the mask information have been alreadyrecorded on the HD, processing at the step 121 is skipped.

The user image data representing the user image is read by the FD drive80 or CD-ROM drive 79, or the user image data is produced upon readingthe photograph or the film, and is temporarily stored in a RAM 74. Thetemplate image data and the mask information are read from the HD, andare temporarily stored in the RAM 74. In a predetermined memory area inthe RAM 74, the user image represented by the user image data or a partthereof is inlaid in the window Aw of the template image represented bythe template image data, referring to the mask information, under thecontrol of a memory controller 85. Consequently, a composite image isproduced (step 122). Composite image data representing the compositeimage is fed to the VRAM 82. The composite image data is read from theVRAM 82 and is inputted to the display device 84 through the DAconverter 83, to display the composite image.

If the composite image which the user desires is obtained, compositeimage data representing the composite image and mask informationcorrelated with the composite image data are transmitted by a modem 81to the image synthesizer 100 from the client computer 71 via the network(step 123).

In the image synthesizer 100, the composite image data transmitted fromthe client computer 71 and the mask information correlated with thecomposite image data are received via the modem 81 (step 131). Thecomposite image data and the mask information which have been receivedare temporarily stored in the RAM 74 in the image synthesizer 100.

In the image synthesizer 100, the template image and the user image areseparated from the composite image by referring to the mask information(step 132). This separation processing can be performed by a processingreverse to the processing in which the composite image is produced usingthe user image, template image and the mask information. On the RAM 74,the template image data representing the template image and the userimage data representing the user image are stored in separate areas.

The user image and the template image are extracted from the compositeimage and are separated from each other in the following manner, forexample. OR logic operation between the composite image and the maskimage produces the template image having the blank portion correspondingto the window Aw. The AND logic operation produces the user image whichhas been inlayed in the window Aw.

The template image data representing the template image which has beenseparated from the user image is subjected to color correction (colorconversion) performed by a CPU 72, as required. The user image datarepresenting the user image which has been separated from the templateimage is subjected to color correction (color conversion) by the CPU 72,as required (step 133). Usually, the color correction (color conversion)performed on the template image data is different from that on the userimage.

Consider a case where (average) brightness of the template image and(average) brightness of the user image are different from each other. Ifthe user image and the template image different in brightness aresynthesized, the produced composite image gives somewhat strangefeeling. Thus, the template image and the user image are separated fromthe composite image in the image synthesizer 100, and the separatedtemplate image and the user image are individually subjected tobrightness correction depending on the brightness of each image so thatthe brightens in both images harmonize with each other as a whole. Thecorrected template image and user image are then resynthesized. Thecomposite image obtained by the resynthesis exhibits balanced appearanceand gives good feeling.

Further, the color correction or conversion may be effected based on theuser's desire. For example, the template image is converted intomonochrome image and the user image is corrected to exhibit sepia coloras a whole. Of course, both of the template image and the user image arenot necessarily subjected to color conversion, but at least one of theimages may be subjected to color conversion.

A composite image is produced again from the template image datarepresenting the template image and the user image data representing theuser image at least one of which has been subjected to the colorcorrection using the corresponding mask information (step 134).

Composite image data representing the composite image obtained by theresynthesis is temporarily stored in the VRAM 82. The composite imagedata is read out from the VRAM 82, and is fed to the display device 84through the DA converter 83. The composite image is displayed on thedisplay device 84, and is confirmed by an operator of the imagesynthesizer 100.

When the composite image is confirmed by the operator, the compositeimage data is fed to the color printer 90A which prints the compositeimage under control of the printer control circuit 89 (step 135).

There is a case where the template image data is down loaded to theclient computer 71 from the image synthesizer 100, and the compositeimage is produced using the template image data in the client computer.The template image data has already been subjected to color correctionin dependence upon the characteristics of the color printer 90A in theimage synthesizer 100. The user image used to produce the compositeimage may have not been subjected to color correction in order to besuitable for the color printer 90A. In such a case, if the user image isseparated from the composite image, the separated user image can besubjected to color correction in accordance with the characteristics ofthe color printer 90 A. The composite image obtained by the resynthesisprocessing using the color-corrected user image has a proper color as awhole.

Although in the above-mentioned embodiment, the client computer 71 andthe image synthesizer 100 can communicate data to each other via thenetwork, the data communication need not necessarily be made possible.For example, a composite image may be produced in the client computer71, and data representing the produced composite image and the maskinformation may be stored in a portable storage medium such as an FD andread by mounting the portable storage medium on the image synthesizer100.

Although the present invention has been described and illustrated indetail, it is clearly understood that the same is by way of illustrationand example only and is not to be taken by way of limitation, the spiritand scope of the present invention being limited only by the terms ofthe appended claims.

What is claimed is:
 1. An image processing apparatus comprising: animage area designating device for designating an area to be extracted ofan original image displayed on a display screen; color changing meansfor changing the color of an area, excluding the area to be extractedwhich has been designated by said image area designating device, of thedisplayed original image into a particular color; and storagecontrolling means for storing pixel data representing pixels havingcolors excluding the particular color obtained by the change in saidcolor changing means in the displayed original image and coordinate datarepresenting positions of the pixels in the displayed original imagewith the pixel data and the coordinate data correlated with each other.2. The image processing apparatus according to claim 1, wherein saidstorage controlling means stores in a storage medium pairs of the pixeldata serving as image data of the pixels composing the extracted imageand coordinate data representing positions of the pixels.
 3. The imageprocessing apparatus according to claim 1, wherein said storagecontrolling means stores the pixel data and the coordinate data uponarranging them for each pixel.
 4. The image processing apparatusaccording to claim 1, further comprising: pixel data compressing meansfor compressing the pixel data, and coordinate data compressing meansfor compressing the coordinate data, said storage controlling meansstoring the pixel data compressed by said pixel data compressing meansand the coordinate data compressed by said coordinate data compressingmeans.
 5. The image processing apparatus according to claim 1 furthercomprising: data reading means for reading the pixel data and thecoordinate data which are stored by said storage controlling means incorrelation with each other; and display controlling means forcontrolling a display device such that the image is displayed bydisplaying the pixels represented by the pixel data read by said datareading means in the positions represented by the coordinate data. 6.The image processing apparatus according to claim 2 further comprising:data reading means for reading, from the storage medium storing imagedata comprising a set of pairs of pixel data representing pixels andcoordinate data representing the positions of the pixels on the image,the pixel data and the coordinate data; and display controlling meansfor controlling a display device such that the image is displayed bydisplaying the pixels represented by the pixel data read by said datareading means in the positions represented by the coordinate data. 7.The image processing apparatus according to claim 4 further comprising:data reading means for reading the compressed pixel data and thecompressed coordinate data which are stored by said storage controlmeans; pixel data expanding means for expanding the compressed pixeldata read by said data reading means; coordinate data expanding meansfor expanding the compressed coordinate data read by said data readingmeans; and display control means for controlling a display device suchthat the image is displayed by displaying the pixels represented by thepixel data expanded by said pixel data expanding means in the positionsrepresented by the coordinate data expanded by said coordinate dataexpanding means.
 8. The image processing apparatus according to claim 4,wherein said storage controlling means stores in a storage medium thecompressed pixel data and compressed coordinate data.
 9. The imageprocessing apparatus according to claim 8 further comprising: datareading means for reading, from the storage medium, the compressed pixeldata and the compressed coordinate data; pixel data expanding means forexpanding the compressed pixel data read by said data reading means;coordinate data expanding means for expanding the compressed coordinatedata read by said data reading means; and display control means forcontrolling a display device such that the image is displayed bydisplaying the pixels represented by the pixel data expanded by saidpixel data expanding means in the positions represented by thecoordinate data expanded by said coordinate data expanding means. 10.The image processing apparatus according to claim 1 further comprisingmeans for subjecting coordinate data which is stored by said storagecontrolling means to coordinate conversion.
 11. An image processingmethod comprising the steps of: designating an area to be extracted ofan original image displayed on a display screen; changing the color ofan area, excluding the designated area to be extracted, of the displayedoriginal image into a particular color; and storing pixel datarepresenting pixels having colors excluding the particular colorobtained by the change in the displayed original image and coordinatedata representing positions of the pixels in the displayed originalimage with the pixel data and the coordinate data correlated with eachother.
 12. The image processing method according to claim 8 furthercomprising the steps of: reading the pixel data and the coordinate datawhich are stored in correlation with each other; and controlling adisplay device such that the image is displayed by displaying the pixelsrepresented by the read pixel data in the positions represented by thecorresponding read coordinate data.
 13. The image processing methodaccording to claim 11 further comprising the steps of: reading the pixeldata and the coordinate data which are stored in correlation with eachother, compressing a set of the read pixel data and compressing a set ofthe read coordinate data, and storing in a storage medium the compressedpixel data and the compressed coordinate data.
 14. The image processingmethod according to claim 13 further comprising the steps of: reading,from the storage medium, the compressed pixel data and the compressedcoordinate data; expanding the read compressed pixel data; expanding theread compressed coordinate data; and controlling a display device suchthat the image is displayed by displaying the pixel represented by theexpanded pixel data in the positions represented by the expandedcoordinate data.